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ABSTRACT 



The ability to understand and frame problems encountered in 



practice is a critical skill for school leaders. This paper reports on a 
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in the Prospective Principals Program at Stanford University, a preparation 
program for school administrators that relies on problem-based learning (PBL) 
strategies for 40 percent of the curriculum. A literature-based definition of 
problem- framing ability is developed, and a theoretical rationale for the 
study is introduced. Incorporating the use of a quasi -experimental study 
design, the program- framing skills in three successive student cohorts, with 
graduated levels of exposure to the PBL, are assessed. Analysis reveals that 
all three cohorts differ significantly in problem- framing ability, associated 
with their level of exposure to PBL. Qualitative data, collected through a 
series of student interviews, support the quantitative findings. 

Problem-based learning experiences are instrumental for developing 
prospective principals' ability to frame problems in practice. Finally, the 
paper discusses the implications of the findings for instructional practice 
and explores ideas for possible future research on the use of problem-based 
learning in administrator preparation. An appendix contains Subskills in 
Problem Framing Ability. (Contains 41 references.) (Author/DFR) 
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Developing the Problem- framing Skills of Prospective 
Principals 



Abstract 

The ability to understand and frame encountered problems in 
practice is a critical skill for school leaders . This paper reports on 
a study that inquired about the teaching and learning of problem- framing 
skills in the Prospective Principals Program at Stanford University , a 
preparation program for school administrators that relies on problem- 
based learning (PBL) strategies for 40% of the curriculum . A 
literature-based definition of problem- framing ability is developed , and 
a theoretical rationale for the study is introduced. Incorporating the 
use of a quasi -experimental study design , the problem- framing skills of 
three successive student cohorts , with graduated levels of exposure to 
PBL , are assessed . An ANCOVA analysis reveals that all three cohorts 
differ significantly in problem- framing ability , associated with their 
level of exposure to PBL. Qualitative data , collected via a series of 
student interviews , supports the quantitative findings . The results 
suggest that problem-based learning experiences are instrumental for 
developing prospective principals' ability to frame problems in 
practice. Finally , the paper discusses the implications of the findings 
for instructional practice and explores ideas for possible future 
research on the use of problem-based learning in administrator 
preparation . 
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Criticisms levied at the preparation of educational 
administrators during the mid-to-late 1980 's yielded several 
institutional responses, comprising alternatives to 
conventional curricular and instructional practices in the 
field (c.f., Bridges with Hallinger, 1992; Milstein, 1993; 
Murphy, 1992; Murphy, 1993; Osterman & Kottkamp, 1993). Many 
of these reform efforts incorporate a pedagogical shift from 
faculty-centered to student-centered approaches that more 
actively involve prospective administrators in the learning 
process (McCarthy, 1999). These approaches, as McCarthy 
(1999) notes: 

encourage the use of inductive, problem-based 
strategies that are grounded in adult learning theory 
and the reality of schools (c.f., Bridges, 1992; 
Hallinger & McCary, 1991; Mulford, 1985; Murphy, 

1992; Shibles , 1988) 

In some instances, new-fashioned approaches to preparation 
have relied upon a contextualized view of the thinking and 
learning process, often termed situated cognition, in which 
knowledge is created and made meaningful by the context and 
activities through which it is acquired (Prestine and 
LeGrand, 1991) . 

One such alternative approach for preparing school 
leaders, problem-based learning (PBL) , originated in medical 
education and is conceived of as a method of instruction and 
an approach to building a curriculum that employs complex, 
interdisciplinary problems taken from professional practice 




as the starting point for learning (Bridges with Hallinger, 
1992). PBL places students in the role of the school 
administrator through the use of contextualized problem 
scenarios, wherein they must work with others to understand 
and solve problems associated with school leadership. 

The research reported in this paper employs a conceptual 
framework rooted in theories of cognition and social- 
psychology to support an inquiry into the teaching and 
learning of problem- framing ability in a problem-based 
administrator preparation program. Specifically, the study 
tests the hypothesis that greater exposure to problem-based 
preparation experiences is associated with greater problem- 
framing ability among prospective principals. 

LEADERSHIP, PROBLEM-SOLVING AND PBL 

Scholars within the field of educational leadership have 
suggested that because of the highly contingent nature of 
school leadership, a focus on improving the quality of 
administrators' problem-solving is likely to be more 
productive in preparing principals than a focus on teaching 
specific actions or behaviors (Leithwood & Steinbach, 1992). 
This direction for preparation seems appropriate, given the 
complexity of dilemmas encountered in the modern 
principalship, and the vast array of snarls that are present 
in leading a school. Successful principals must be skilled 
in the ability to understand, formulate, and solve problems. 

As Achilles & Hoover (1996) note elsewhere, a number of 
scholars have illustrate differences in the nature of 
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problems and the ways they are encountered. Getzels (1979, 
1986) differentiated between presented problems, those with a 
known formulation, and discovered or created problems, those 
that must be formulated or invented by the problem-solver. 
Similarly, Wagner (1993) distinguished between academic and 
practical problems. Perhaps most usefully for the purpose of 
this study, Leithwood (1995) differentiated between routine, 
well-structured problems, and ill-structured, non-routine 
problems. School administrators regularly encounter problems 
in their work environment that are both routine and non- 
routine . 

Problem-based learning specifically intends to 
familiarize prospective principals with various types of 
problems they will face in their future roles, and to develop 
skills of understanding and solving problems (Bridges with 
Hallinger, 1992). Given these intentions, knowledge about 
learning outcomes in the area of problem-solving resulting 
from experience with problem-based learning has a high degree 
of relevance for the field. However, limited empirical 
evidence exists at present regarding any outcomes associated 
with problem-based preparation in educational administration. 
This study examines one intended outcome of problem-based 
preparation, an aspect of prospective principals' problem- 
solving skill, the ability to understand and frame problems. 
PROBLEM- FRAMING ABILITY 

Clinical reasoning in medicine, as well as problem- 
solving behavior in educational administration, can be 



characterized as a creative process in response to a 
problematic situation in which the initial framing of the 
problem is fundamental to the development of a useful 
solution. John Dewey (1910) noted that "the essence of 
critical thinking is suspended judgment; and the essence of 
this suspense is inquiry to determine the nature of the 
problem before proceeding to attempts at its solution." Dewey 
uses the example of a medical doctor's diagnosis of a 
patient's ailment. He notes: 

Imagine a doctor is called in to prescribe for a 
patient. The patient tells him some things that are 
wrong; his experienced eye, at a glance, takes in 
other signs of a certain disease. But if he permits 
the suggestion of this special disease to take 
possession prematurely of his mind, to become an 
accepted conclusion, his scientific thinking is by 
that much cut short. A large part of his technique, 
as a skilled practitioner, is to prevent the 
acceptance of the first suggestions that arise; even, 
to postpone the occurrence of any very definite 
suggestion till the trouble -- the nature of the 
problem — has been thoroughly explored (Dewey, 1910, 
pg. 74) . 

The conception of an initial definition of the problem 
is widely supported, and perhaps by none more eloquently than 
Jacob W. Getzels (Getzels, 1979; Getzels & Csikszentmihalyi , 
1976) . Getzels studied the creative process in visual artists 
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and refers to the initial formulation of a problem as 
"problem- finding" ability. Getzels found that a person who 
has learned to be concerned with problem- finding will not 
approach a problem situation with a ready-made solution in 
mind. A skilled problem- finder will not let past experience 
completely determine what is to be done; rather, he or she 
will let the new challenge suggest new solutions. 

The research reported herein focused only on students' 
initial understanding and formulation of encountered 
problems. Within the field of medicine, the comparable first 
step is the formation of a diagnosis which does not specify 
treatment. Specifically, this preliminary aspect of problem- 
solving skill is defined here as being consistent with 
Cuban's (1990) notion of problem- framing ability. Cuban 
notes : 

Framing a problem, then, is a subjective process. It 
depends upon one or more facts that show a 
discrepancy between what is and what ought to be. It 
depends upon the perceptions of the person or group 
who interpret the data and do the defining. What 
shapes (these perceptions) are (one's) previous 
personal and work experiences, (one's) beliefs and 
values, the position (one holds) within an 
organization, and the expected role (one) is to play 
within that organization. (Cuban, 1990, pg. 2) 

Bolman and Deal (1993) point out that problem- framing is 
no easy task in the modern organization, and the manner in 
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which a problem is framed determines the script that 
ultimately guides action. They note: 

Leaders in particular are required to make sense of 
ambiguous, complex, and puzzling events. When they 
frame accurately and respond appropriately, puzzles 
and problems become promising opportunities. When 
frames distort or overlook essential elements of a 
situation, leaders "lose the bubble , " feel out of 
control, and fall back on familiar scripts even if 
their actions only make things worse. (Bolman and 
Deal, 1993, pg. 23) 

They go on to suggest that the leader's ability to reframe a 
problem involves a conscious effort to size up a situation 
using multiple lenses (pg. 24) . Bolman and Deal view the 
ability to reframe complex problems as an important 
precondition to the effective exercise of leadership. 

As one product of the problem-solving research done by 
Leithwood and various colleagues, Leithwood and Stager (1989) 
developed a theoretical model comprised of six grounded 
components that were identified in school principals problem- 
solving processes. For each of these six categories, 

Leithwood et al . identified a more specific set of subskills 
associated with the behavior, as well as performance 
indicators that differentiate between "expert" and "novice" 
problem-solvers . 

An analysis of the Leithwood and Stager (1989) 
framework, suggests that a number of these various subskills 
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focus directly on abilities or processes employed in the 
initial formulation or framing of administrative problems, 
that are consistent with the skills of problem- framing taught 
in problem-based learning. For example, from Leithwood et 
al . , expert principals (a) have an understanding of the 
importance of developing a clear interpretation of a problem, 
(b) have the ability to develop a clear interpretation of the 
problem and the ability to describe this interpretation to 
others, (c) seek out and take into account the interpretation 
others have of the problem, (d) carefully check their own 
assumptions relative to others' interpretation of problems, 

(e) have less of a personal stake in any preconceived 
solution than non-experts, (f) anticipate obstacles likely to 
arise during group problem solving, and (g) plan in advance 
for how to address anticipated obstacles. 

Larry Cuban (1990) highlights another critical subtask 
inherent to the problem- framing process. Cuban notes that 
problems encountered by principals in practice frequently 
have solutions already embedded in them. Leaders often 
commit the fatal error of strongly embracing a preconceived 
solution before a problem has been clearly defined and 
understood. This suggests that an additional important step 
in problem- framing is the principal's ability to recognize a 
problem that has been presented with a predetermined 
solution, and then to reframe the problem in solution-free 
terms. Different and equally sound solutions to a problem 
may exist, particularly with regard to non-routine, complex 
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dilemmas. By developing a clear understanding and definition 
of a problem situation, the principal creates the frame in 
which all viable solutions can be considered. Indeed, the 
way a problem is framed fundamentally determines whether a 
predetermined solution will quickly follow, or other 
alternative solutions will be considered. Appendix A provides 
a rubric of these subskills that serves as a definition of 
problem- framing ability for the purposes of this research. 
PROBLEM- FRAMING ABILITY IN PRACTICE 

An illustrative example may help to clarify how these 
subskills act in concert in the problem- framing process. In 
framing a problem, one must first understand the problem as 
presented. School administrators encounter problems that 
reside at various levels within a school organization. The 
way a problem is initially presented frequently reflects the 
particular bias or understanding of the individual or group 
who has raised the issue. 

Imagine a situation in which a teacher approaches the 
principal with a complaint about a particular student's 
behavior in class, and demands that the student be removed 
and placed in another class. Initially, it is important for 
the principal to understand that the teacher sees the problem 
as residing with the student. 

Consistent with this understanding, a principal 
employing expert skills in problem framing would recognize 
that the teacher has framed the problem with a preconceived 
solution, and this recognition would trigger a reflective 
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process, however brief. From the teacher's point of view, 
the appropriate solution, already embedded, is to move the 
student into another class. A skilled principal would 
understand that the way the problem is presented lends itself 
to only one solution, and would endeavor to look deeper into 
the nature of this problem in order formulate a clear, 
unbiased interpretation. 

The principal may have some personal views about the 
teacher's interactions with students based upon previous 
encounters of this type. This knowledge might raise some 
assumptions that need to be checked against the facts in this 
particular case. An expert principal will not overlook the 
possibility that the way the teacher has framed the problem 
may be absolutely correct, but also will be careful to use 
what he or she knows in reflecting on the situation. 

Given the recognition that a pre-determined solution has 
been presented in the situation, the principal might choose 
to reframe the problem in such a way that various other 
useful solutions could be considered. Rather than placing the 
blame on the student or on the teacher, the principal might 
search for alternative ways of framing the situation. The 
principal may rely on an established problem-solving process 
for working through the situation -- collecting more 
information, collaborating with support staff who may have 
insights into the situation, or considering other lenses on 
the problem. For example, the complainant may be a teacher 
who has never experienced this kind of difficulty with a 
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student, or this may be a teacher who has a history of many 
of these types of encounters. The student in question may 
have a long record of behavioral difficulties that limit the 
options available. Or, the student may have no previous 
record of problems in the classroom. 

Depending on the situation, the expert principal would 
anticipate obstacles that are likely to arise from various 
solution alternatives. Taking a full account of the facts in 
the situation would enable the skillful principal to look 
ahead and figure out how the obstacles can be addressed 
should they arise. 

While this simplistic example does not do justice to the 
intricate interrelatedness of the various subskills in 
problem- framing, it is offered to provide a sense of the 
thought processes a principal skilled in framing problems 
might use when encountering such a situation on the job. In 
practice, this process would obviously occur rapidly and 
naturally for a principal who possessed a high degree of 
facility in this area. 

THEORETICAL FRAMEWORK 

The conceptual warrant for inquiring about a 
relationship between problem-based instruction and student's 
ability to understand and frame problems of practice can be 
established several ways. Bridges (1992) developed a 
rationale for the use of problem-based learning in the 
preparation of educational administrators grounded in three 
different conceptual areas - theories of cognition, 
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motivation, and function. A brief review of the cognitive 
perspective is included here, as it is most clearly 
suggestive of a link between problem-based learning 
experiences and students' acquisition of problem- framing 
skills. The discussion of the cognitive rationale is 
followed by the introduction of a second theoretical 
perspective drawn from social psychology which further 
supports the inquiry. 

Bridges (1992), drawing on the work done in medical 
education by Schmidt (1983), notes three conditions created 
within a problem-based learning environment that information 
theory links to subsequent retrieval and appropriate use of 
new information: activation of prior knowledge, encoding of 
knowledge in a specific context, and opportunity to elaborate 
on that information. 

First, in problem-based learning prior knowledge is 
activated. Students are expected to exercise knowledge they 
already possess in order to understand the problem situation 
they face. Bridges (1992, p. 9) notes that "this prior 
knowledge and the kind of cognitive structure in which it is 
stored determine what is understood from the new experience 
and what is learned." 

Second, in PBL new knowledge is encoded in a context 
modeled on practice. Research on cognition suggests that 
knowledge is more likely to be remembered or recalled in the 
context in which it was originally learned (Godden and 
Baddeley, 1975). As Bridges notes (1992, p.9) "encoding 
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specificity in problem-based learning is achieved by having 
students acquire knowledge in a functional context, that is, 
in a context containing problems that closely resemble the 
problems they will encounter later on in their professional 
careers . " 

Third, PBL offers students the opportunity to elaborate 
on information that is learned. Elaboration provides 
redundancy in memory, which in turn reduces forgetting and 
abets retrieval (Bridges, 1992, p.9). One might expect that 
the skills of problem- framing are better understood, 
processed and recalled through strategies such as small group 
discussion, peer review of ideas about how particular 
information applies to a given problem, and the practice of 
reflecting back on problem-solving processes through 
debriefing practices and personal essays of what was learned. 

In addition to the cognitive rationale outlined above, 
social psychological theory also provides conceptual support 
for an inquiry into links between modes of instruction and 
the learning of various skills. Social comparison theory 
(Festinger, 1954) posits that humans have an innate need to 
evaluate their own abilities with those of similar others. 
Given that students in the preparation program entered with 
the common intention to pursue a career in school leadership, 
one might expect comparisons of abilities central to that 
work to be prevalent. Furthermore, Rosenholtz and Simpson 
(1984) suggest that classrooms which are organized to 
recognize and promote multiple performance dimensions will 



provide multiple bases for students to compare and evaluate 
their abilities. 

Designers of problem-based learning implicitly recognize 
that social comparison of ability is prevalent and central to 
students' appraisal of self and others. This is evident in 
the incorporation of performance-based assessments, 
assignment of various roles to students in the problem- 
solving process, emphasis on the practice of a variety of 
"real world" skills important for the work of the principal, 
including the framing of administrative problems, and in the 
open, often public, nature of the feedback processes used in 
PBL . One might expect greater opportunities for ability 
comparison and self-evaluation to result in stronger ability 
within students. 

METHODOLOGY 
Sub j ect s 

The subjects included in this study were eighteen 
students enrolled in the Stanford University Prospective 
Principals Program. The specific students included in the 
study comprise three cohorts admitted to the program in 
successive years (n=six students per cohort) . An extensive 
set of background data, including age, gender, ethnicity, GRE 
scores, years of teaching experience, and level of prior 
education, was collected on each student prior to beginning 
the study. The nature and use of the background measures is 
introduced in the analysis. 

Study Design 



Assessment of students' problem- framing ability was 
accomplished through the use of a quasi-experimental , post- 
test only design (Cook & Campbell, 1979) . Given the 
university setting, and the cyclical nature of turnover in 
the Prospective Principals Program, the study is further 
categorized as a cohort design in a formal institution with 
cyclical turnover (Cook et al . , pg. 126) . Cook and Campbell 
note that cohorts are useful for experimental purposes 
because (a) some cohorts receive a particular treatment while 
preceding or following cohorts do not, and (b) it is often 
reasonable to assume that a cohort differs only in minor ways 
from its contiguous cohorts (pg. 127) . 

The purpose of the design was to test whether the three 
successive PPP cohorts differed in their problem- framing 
ability. The cohorts were tested after receiving different 
levels of exposure to problem-based learning. A schematic 
representation of the study design, including grand mean 
scores of each cohort, is presented in Table 1. 

<Insert Table 1 here> 

Each of the eighteen subjects was individually presented 
with a series of five short, written administrative problem 
scenarios that were developed specifically for the purpose of 
this study. All represented actual problems that were faced 
in practice by a school principal. While not explicitly known 
by the subjects, each scenario featured an embedded solution 
as part of the problem formulation. Students were asked to 
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respond in writing to each of the following questions about 
each problem scenario : 



1. How has the problem been defined in this scenario? 

2. Employing what you know and believe about solving 
problems in practice, reflect on how this problem 
has been framed. Be as thorough as possible. 

3. If faced with this situation in practice, would 
you reframe the problem? If so, how? (If you 
choose to restate the problem, justify your 
reasons for doing so.) 



Cohort 1 (first-year group), completed the exercise prior to 
participation in any problem-based learning experience, nor 
any program exposure to the concept of problem- framing. 

Cohort 2 (second year group) completed the assessment during 
their second summer in the problem-based practicum, after 
exposure to approximately seven (7) PBL projects; Cohort 3, 
during their final summer in the program, after exposure to 
approximately twelve (12) PBL experiences. 

All three groups completed the scenario exercises under 
similar conditions. Students responded in writing to the 
five scenarios, in a consistent order, during one sitting in 
the same university classroom. Students were given little 
instruction about the exercise other than to read each 
scenario and respond in writing to the three questions. 
Students were not allowed to discuss their responses to the 
scenarios until after all materials had been turned in by all 
students. Responses were collected in a manner that insured 
a blind scoring process. 

Measurement 
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The subskills of problem- framing included ten 
indicators, in three categories, and formed the basis for 
construction of a scoring instrument used in assessing 
responses to the problem scenarios. 1 Three (3) independent 
raters assessed student responses to the three questions 
using the ten item scoring instrument. All three raters had 
experience as principals, and one rater was not connected to 
students nor to the preparation program in any way. All 
responses were scored blind. All raters scored each 
subject's written responses on a 0-3 scale (0 = no 
evidence of indicator; 3 = strong-compelling evidence of 
indicator) corresponding to each indicator. 

Reliability and Validity 

The reliability of the measure used to assess the 
prospective principals' problem- framing ability was analyzed 
using generalizability (G) theory, a statistical theory about 
the dependability of behavioral measurements (Shavelson & 
Webb, 1991) . To briefly summarize, the G study examined 
variablity in scoring due to persons (p) , raters (r) and 
scenarios (s) , and all possible combinations of those three 
factors. This type of analysis enables one to determine the 
sources of variability within the measure, their relative 
magnitudes, as well as the overall reliability 
(generalizability) of the measure. It was revealed that, as 
expected, differences between persons (subjects) accounted 
for a relatively high magnitude of the variance in scores. 
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approximately 44%. Magnitude of variability due to raters 
was comparatively very small (2%), as was variability across 
the five problem scenarios (0%) . Furthermore, the 
generalizability coefficient generated for the measure, 
across raters, scenarios, and subjects, was on the order of 
.89, indicating the measure was highly reliable. 

Of particular interest for the purposes of this study is 
the content validity of the problem scenarios. Content 
validity concerns the extent to which the items employed 
adequately sample the domain or concept of interest; it 
examines whether the items are representative of the domain 
of interest. Ideally, content validity should be established 
deductively "by defining a universe of items and sampling 
systematically within this universe to establish the test" 
(Cronbach & Meehl , 1955, p. 282). Such is not always the 
case, however, particularly in social science research when 
definition of a universe of content is difficult. Instead, 
the case for content validity is made inductively here, based 
upon information regarding the construction of the problem 
scenarios . 

Consistent with the work of Wigdor and Green (1991) , the 
content validity of the problem scenarios was assessed along 
two axes: abstractness , and fidelity of the measure. The 
abstractness of the measure assessed the likelihood that a 
principal might actually confront the problems detailed in 
the measure. A highly concrete measure, in this case, would 
include highly relevant problems, such as those a school 
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administrator might reasonably expect to encounter in 
practice. The more abstract the measure, the less it would 
appear to approximate an actual administrative problem. The 
fidelity of the measures is defined as an assessment of the 
extent to which the measure, as encountered by the students 
in the study, mirrors the real-world situation of interest. 

A high fidelity problem scenario would closely replicate the 
manner in which an administrator encounters it in practice. 

The problem scenarios used in the study were developed 
by the researcher from actual experiences encountered by 
school administrators, including three problem scenarios 
encountered in practice by the researcher himself, one 
scenario encountered by a colleague, and one scenario adapted 
from a set of materials developed by a national principal's 
organization (NASSP) . Moreover, while each problem was 
presented in a specific context, the underlying nature of the 
problems cut across geographic, experiential and grade level 
boundaries. The issues included an interpersonal conflict 
among staff members, a classroom management situation, a 
racially-charged school site council decision, a situation 
involving a district-level curricular adoption, and a freedom 
of speech issue. Each represents a concrete domain of 
interest for prospective school administrators and, while not 
exhaustive of all the domains, are representative of those 
commonly encountered. That two outside analysts, each with 
experience as principal, found the measure to be "face valid" 
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further supports this claim. Abstractness of the measure, 
therefore, was assessed to be relatively low. 

The open-ended nature of the questions that students 
answered about each scenario more closely approximated 
reality than, for example, choosing from a list of pre- 
determined responses in multiple-choice format. Students 
formulated their own unique responses to each scenario. 
Whether the formulation of a response culminates in verbal or 
written discourse may be less important. If anything, the 
process of writing provided students an opportunity for 
additional mental rehearsal of their responses, and may 
thereby strengthen the responses. Given this, it was 
determined that the measure was of moderate fidelity. A 
method of measurement with higher fidelity, perhaps involving 
role players or some other manner of face-to-face interaction 
rather than a pencil and paper exercise, would have proven 
difficult to convene given the constraints of the testing 
situation. This said, the fidelity of the problem scenarios 
was recognized as a limitation in the study design. 
Qualitative Data Collection 

In addition to the quantitative study, a series of 
interviews were conducted with all eighteen students. While 
the interview protocol focused primarily on another aspect of 
the study not reported here, aspects of the interviews were 
used to elicit feedback from students regarding the 
development of their administrative abilities. Some student 
responses related information about their development of 
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problem-solving skills. These data provide a secondary focus 
in the analysis and a few illustrative student comments are 
included in this paper as a means for triangulating with the 
results found in the quantitative study of problem- framing 
ability. 

ANALYSIS 

To attack the problem of separating the effect of the 
treatment from the possible effects of selection differences, 
a two-step statistical analysis was conducted. First, a 
correlation matrix was created to determine the nature of the 
relationships between a number of background measures and the 
dependent variable, problem- framing ability. A composite 
background measure found to be highly correlated with the 
dependent variable was identified for use as a covariate in 
further statistical analysis. Second, an analysis of 
covariance (ANCOVA) was conducted to assess differences 
between cohort groups on problem- framing ability, employing 
the composite background measure as a covariate. Detailed 
explanation in each area follows. 

Correlation 

In order to determine the nature of the relationships 
between available background measures and the dependent 
variable, a correlation matrix was calculated that included 
age, gender, ethnicity, GRE scores (verbal, quantitative and 
analytic, and a composite GRE variable -- GREAQ) , years of 
education-related work experience, subjects' level of prior 
education, as well as each subjects' grand mean of problem- 
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framing scores. The correlation matrix is presented as Table 

2 . 

dnsert Table 2 here> 

Scores on all three of the GRE subtests (verbal, 
quantitative and analytic) had a significant positive 
correlation with the dependent variable, problem- framing 
ability (.5085, .6918 and .6145, respectively). The 

quantitative and analytic subtests were most highly 
correlated with problem- framing ability (alpha c.Ol) . A new 
variable, called GREAQ, was created by summing subjects' 
scores from these two tests. GREAQ was also highly 
correlated with the dependent variable (.6895, significant at 
alpha < .01). Subjects' gender also showed a moderate 
negative correlation with the dependent variable (-.4678, 
alpha < .05). However, this significant correlation can be 
explained by the fact that all the subjects with no exposure 
to problem-based learning were female. Differences in the 
problem- framing scores between males and females with some 
level of exposure to PBL were not significant. 

Analysis of Covariance 

Subjects' responses on the five problem- framing 
scenarios were scored by three independent raters, using the 
ten item scoring instrument. A grand mean score, across all 
items (10), scenarios (5) and raters (3), was calculated for 
each subject. This grand mean represents the closest 
approximation available to a "true" score for each subject's 
problem- framing ability, as measured by the instrument. Given 
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that the general izability analysis was conducted on the 
complete design (persons x raters x scenarios) , and 
reliability of the measure was established at the level of 
the grand mean, the analysis of the dependent variable was 
appropriately conducted at that same level . 

An analysis of covariance (ANCOVA) was calculated on 
subjects' grand mean of problem- framing scores by cohort. 
ANCOVA uses a covariate, an individual difference variable 
which is highly correlated with the dependent measure, as a 
statistical control to reduce unexplained error variation. 

To do this, in ANCOVA the dependent variable is adjusted 
statistically to remove the effects of error variation 
represented by predictable differences within groups -- 
variation due to the covariate (Ruiz-Primo, Mitchell, and 
Shavelson, 1996) . Given the small n, and the fact that, for 
each covariate employed in the ANCOVA one degree of freedom 
is lost from the error term, a decision was made to include 
only one covariate, GREAQ, in the model. 

The research hypothesis proposed that the ability to 
frame administrative problems is greater among prospective 
principals with more exposure to problem-based learning than 
for those who have had less exposure. The ANCOVA summary 
table is presented as Table 3. Two rows in the ANCOVA table, 

clnsert Table 3 here> 

Main Effects and Explained, can be ignored. The former 
summarizes the combined main effects; the latter summarizes 
all systematic effects in the study. The remaining rows in 
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the ANCOVA table include the main effect for Cohort (length 
of exposure to PBL) , the effect for the covariate GREAQ, the 
error term (residual) , and a row summarizing the total of all 
sources of variation. 

A significant main effect was found for Cohort (Fc.OOO), 
representing the levels of exposure to problem-based 
learning. This result indicates that significant mean 
differences were present among the cohort groups on the 
dependent variable, problem- framing ability. Second, it is 
important to note that the error term (residual) was reduced 
by more than half as compared to a simple one-way ANOVA 
without the covariate. 2 Finally, as expected, the covariate F 
value was significant (Sig of F- . 002). 

To summarize, the ANCOVA provides support for the 
research hypothesis, concluded that significant differences 
among treatment group means were found. However, which group 
means differ significantly cannot be determined from the 
initial ANCOVA table. Post hoc comparisons of the adjusted 
group means were conducted to make this determination. 

Post Hoc Comparisons of Adjusted Means 

The significant main effect revealed in the ANCOVA is 
based on comparison of the adjusted Cohort means for the 
dependent variable. In order to conduct post hoc comparisons 
between groups. Cohort means were statistically adjusted to 
remove the effect of the covariate. 3 It was expected in the 
study design that Cohort 3, the group with the most exposure 
to problem-based learning, would score higher on the problem- 
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framing measure than Cohort 2, the group with the next 
highest level of exposure to PBL. Also, it was expected that 
both Cohort 3 and Cohort 2 would score higher than Cohort 1 , 
which had no exposure to PBL. The expected pairwise 
relationships were formulated as three separate null 
hypotheses, using the adjusted means for each cohort. In 
order to test these specific hypotheses, and to determine 
which (if any) of the observed differences were due to 
chance, Tukey's HSD test was used to conduct the pairwise 
comparisons . 4 

All three pairwise comparisons revealed significant 
differences between groups on the dependent variable, 
problem- framing ability. The difference in the adjusted 
means between Cohort 3, those students with the greatest 
exposure to problem-based learning, and Cohort 1, those 
students with no exposure, was significant at alpha < .01. 

The difference between the adjusted means for Cohort 2 and 
Cohort 1 was also significant at alpha < .01. The comparison 
between Cohorts 3 and 2 , who had received graduated levels of 
exposure to PBL, revealed a significant difference at the .05 
level of alpha for greater exposure to PBL. 

Qualitative Findings 

Analysis of qualitative data collected via student 
interviews provided insights about how students viewed 
aspects of their personal problem-solving ability, and how 
this ability had developed. First year students were 
interviewed toward the end of their first summer in the 
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administrator preparation program; second year students 
during their second summer; and third year students near the 
conclusion of their preparation experience. Fourteen of 
eighteen students stressed aspects of problem-solving ability 
as a personal strength compared with others in the field of 
educational administration. This was particularly true of 
responses from students in the second and third year cohorts. 
All of the fourteen who noted problem-solving skills as a 
personal strength attributed the development of these skills, 
at least in part, to their preparation in problem-based 
learning. One third year student offered this assessment of 
how his entire cohort had grown in their ability to frame and 
solve problems over the course of their experience in the 
PPP : 

I think in terms of skills, we're far more adept 
than we were at the beginning of the program at 
the... problem-solving process. As well as the, sort 
of, perspectives that we bring to the problems. I 
think that the program has taught us to consider more 
broadly, the issues that might impact a decision that 
we make. Or the impact that our decisions might 
make, as we're designing solutions. And I think 
that ... that ' s pretty dramatic, right across the 
board. That people have.... internalized the need to 
think broadly in terms of the .... constituencies that 
you need to take into account as you ' re making 
decisions. In a way that we didn't right off the 
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bat. [Student #6285: Text Unit 120] 

A second year student remarked: 

(I'm better at) working through problems because of 
the (PBL) practicum, again. Rather than feeling 
overwhelmed, looking at it. . . . (so) that there’s a way 
to break it down and to work through it. Which is 
something that's been so intimidating bef ore ... Before 
it was like "Oh my God, we've got this huge problem 
and what are we going to do . " And everybody ' s kind 
of frustrated together. But, to actually have a 
process that you can work through. [Student #2002: 
Text Unit 71] 

Another second year student provided an example of how 
problem- framing skills, learned and practiced in the PPP 
practicum, impacted his understanding of decision-making 
processes observed at work: 

I sat down in meetings last year with (organization 
for whom he worked) where. . . . (person's name) , who 
founded the organization came in and said, 

"Okay We need to change our selection model." In 

other words, how we select people. (The leader said) 
"What do you guys think? How should we do that?" 
So.... there were ten people sitting around the table 
saying, "Well, we should do this. We should do that. 
We should do this." And I raised my hand and said, 
"That all sounds great, but what is the problem? 

What exactly is the problem?" And I think that 
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that’s the question that I kent ™. y, • 

at I kept pushing on people all 

ySar ' tG thS P ° int Where 1 they were frustrated 

with me, because I was questioning Just about 

everything we did. But what I sensed was people 
always getting around the table with their own set of 
criteria... the reasons why they think this is a good 
answer, but no common agreement on what the problem 
And so. .. .1 was constantly pushing people to 
spend the time. . . .analyzing what the problem was 
[Student #1029: Text Unit 8 ] 

discussion 

that e J‘ antltatlVe “alysis presented two challenges, one 
resided outside of the control of the researcher, and 
one that was statistically controlled. With regard fco ^ 

ormer, given the small n of each cohort group, only large 
ects sizes would register as statistically significant 
erences between groups. The smaller the sample, after 

' 9reater the Chance -re subtle effects are 
masked in the analysis- this i e * 

• his is the fundamental problem of 

using parametric statistics with small samples. The use of 
non-parametric statistics for analysis was considered 
y but then set aside, predicated on the tenuous 
JPectation that large effects existed which, if found, would 
he suggestive of greater practical significance. Had the 
™ returned a non-significant result, a non-parametric 

data 73 " 3 WOUld haVe been entertained aS a back “P look at the 
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Secondly, a composite background variable was identified 
that was highly correlated with the dependent variable. 
Without removing the effect of the correlated variable, a 
finding of statistically significant differences between 
groups opens the door to alternative interpretations; for 
example, that the exercise was really measuring differences 
in cohort GRE, rather than differences in problem- framing 
ability. However, the inclusion of the correlated variable as 
a covariate in the statistical model increases the power of 
the statistical analysis (Light, et al . , 1990), statistically 
removing the influence of highly correlated GRE scores on 
variances in the dependent variable. 

The results suggest that, within this particular 
preparation setting, greater exposure to problem-based 
learning is associated with greater problem- framing ability 
among students. Qualitative data collected in student 
interviews corroborates this finding. Common sense, as well 
as scholarship in the field of educational administration 
(c.f., Hallinger, Leithwood and Murphy, 1993), suggests the 
strong need for principals to be able to understand, frame 
and solve problems they encounter in practice. To date, 
evidence about the impact of the strategies encompassed by 
problem-based learning on the development of associated 
leadership skills has been thin. The case for using PBL in 
educational administration has been made primarily through 
the study of PBL in medical education; however, research into 
the relationship between medical education models of problem- 
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based learning and students' problem-solving ability is 
characterized by unsystematic study and equivocal evidence. 

As Leithwood et al . (1993) point out in discussing a future 

research agenda involving problem-based learning in the 
preparation of school leaders, educational administration 
"cannot rely on extrapolations from weak evidence, collected 
outside the field, for instructional guidelines" (Leithwood, 
Hallinger and Murphy, pg. 277). While the results found in 
this study of PBL are tempered by various limitations, 
addressed in the next section, the findings also appear to 
merit a discussion of implications for instructional practice 
and curricular planning in educational administration. 
Limitations 

Methodological limitations include the unique character 
of the sample. The unusual nature of the student population 
in the PPP, and the intentionally intimate structure of the 
program also constrain one's ability to generalize to other 
populations. These constraints were identified as limiting 
factors early in the process of conceptualizing the study. 

The presence of these factors detracts from the 
general izability of the findings, even though the instrument 
used to measure problem- framing ability was found to be 
highly reliable. 

Lack of random assignment to groups also limits the 
interpretation of the results. Successive cohorts admitted to 
the PPP go through an identically rigorous admission process, 
and have many similar characteristics, as the exploration of 
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several background characteristics revealed. However, there 
are undoubtedly other background characteristics not examined 
here that may have introduced selection bias not accounted 
for in the analysis. Though one can make the case that the 
use of the ANCOVA is more precise than a statistical 
procedure that ignores differences in background measures 
altogether, it is still likely to be biased. The sameness of 
the cohorts cannot be unequivocally determined on the basis 
of the selected background measures alone. Nonetheless, the 
use of the ANCOVA model rests on this tenuous assumption, and 
therefore tempers the findings. 

One might also question to what extent the findings can 
be attributed to factors other than exposure to problem-based 
learning that influenced students' ability to "think like an 
administrator" over time. Were forces other than PBL at work 
in students' environments in or out of preparation that 
inculcate this kind of thinking? Students in the second and 
third summers of the administrator preparation program may 
have benefited from increased exposure to problems in the 
workplace that they encountered during the intervening 
academic years working as teachers, and the accompanying 
opportunities to practice problem- framing skills learned 
during that time. If this occurred, and if so, how it 
occurred, is not known and therefore leaves open an 
alternative explanation for the findings. 

Finally, as noted previously, the fidelity of the 
problem scenarios was judged to be moderate. While the 
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content was highly concrete and germane to the work of school 
administration, the manner in which the problem scenarios 
were encountered was, in fact, different from the usual ways 
problems are encountered in practice, and so raises questions 
about their validity. 

Implications for Curriculum Development and 
Instructional Practice 

Limitations notwithstanding, the results of the study 
suggest that under particular conditions, an important 
prerequisite of administrators' problem-solving ability, the 
framing of administrative problems, can be successfully 
taught and developed in students during the process of 
preparing for the principalship . While global 
generalizations about the efficacy of problem-based learning 
are not appropriate given the nature of the study design, a 
discussion of the preparation setting and use of problem- 
based learning may be useful to those concerned with teaching 
problem- framing skills to prospective principals. 

First, the application of problem-based learning in the 
Prospective Principals Program accounts for 40% of a total 
administrator preparation curriculum. That curriculum also 
features a series of core courses, specifically designed for 
principal preparation, as well as a particular set of field 
experience projects. Together with the problem-based 
practicum, these three program components play out over the 
course of three consecutive summers of study, and two 
intervening school years . 
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Second, the study highlights a specific version of PBL, 
and the processes used by instructors within that version. 
While literatures from various fields are rife with 
interpretations and uses of the term "problem-based 
learning," Bridges' (with Hallinger, 1992) version of PBL 
offers some clarity regarding background and rationale, 
choice of content, structure and sequencing of projects, the 
role of the instructor, and other specific aspects that 
pertain to the preparation of school leaders . 

Third, as suggested in the conceptual framework, 
instructional practices that activate prior knowledge and 
situate learning in contexts similar to those that will be 
encountered in practice appear to foster the development of 
students' ability to understand and frame problems. 

Moreover, the incorporation of debriefing techniques that 
encourage students' elaboration of knowledge and reflection 
on learning may help solidify a way of thinking about 
problems . 

Finally, the qualitative findings suggest that social 
comparison processes appear to be a factor in shaping and 
developing students' skills and abilities. The research 
further suggests that careful sequencing of skill-based 
expectations through increasingly complex and diverse 
problem-solving experiences may be a useful technique for 
building student skills in problem-f raming . Additionally, by 
focusing on global skills (management and organization of 
meetings, problem framing, written communication, oral 
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communication, time management, etc.) early in preparation, 
students may develop a useful tool kit for approaching ever 
more complex problem situations later on. 

Implications for Future Research 

The results of this research effort provide initial 
evidence, in one specific skill area, with one group of 
students, of the efficacy of a specific method of problem- 
based learning for teaching problem- framing skills to 
prospective principals. The findings signal a potentially 
rich research agenda focused on the study of associated 
outcomes of problem-based preparation. Two possibilities for 
future research stemming from this study are discussed in the 
paragraphs that follow. 

First, a followup study that examines the development of 
problem- framing ability within a PBL setting across time, 
within in one group of students, would provide an opportunity 
to test the findings from this study. Rather than employing 
a quasi -experimental design to compare "snapshots" of 
problem-framing ability across three cohorts, a longitudinal 
study employing a test-retest design would enable comparisons 
to be made within the same subjects. In fact, a follow-up 
examination of the first year PPP cohort is planned, to 
reassess their problem- framing skills at or near the end of 
their final (third) year in the PPP. These data would 
provide one means for assessing the development of problem- 
framing ability over time. 
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The findings also raise the question of whether problem- 
framing ability associated with problem-based preparation 
actually transfers into the early practice of school 
administration. As Bridges and Hallinger (1993) suggest, 
researchers may find it fruitful to examine the extent to 
which graduates of PBL-based programs spontaneously use the 
general problem-solving strategies they acquired during their 
training. Given the findings on problem- framing ability, one 
might use qualitative methods to examine the use of that 
specific skill in the work practice of new school principals, 
who had apprenticed in a problem-based learning environment . 

NOTES 

1 As listed in the Appendix, the indicators derive from Leithwood’s and colleagues’ work on expert-novice 
problem-solving among school administrators (Leithwood and Stager, 1989), and Cuban’s conception of 
problem-framing skill (Cuban, 1990). 

2 A one-way ANOVA comparing problem-framing scores by cohort revealed a residual (error) term of .769. 

3 The statistical formula for calculating the adjusted means comes from Shavelson (1996, pg. 516). 

4 Tukey’s HSD (honestly significant difference) test calculates an observed q value for each comparison, 
which is measured against a critical q value from a generalized studentized range distribution (c.f., 
Shavelson, 1996, pg. 518) to determine whether group differences are significant. 
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APPENDIX A 



Subskills in Problem Framing Ability 



(I) Definition of the stated problem. 

Descriptor : 

• Clear recognition of the stated problem. 

(II) Reflection on the stated problem. 

Descriptors : 

• Identification of the importance of formulating a 
clear interpretation of the problem prior to 
considering possible solutions. 

• Identification of the importance of approaching a 
problem without holding to a preconceived solution. 

• Raising and checking personal assumptions about the 
problem situation. 

• Consideration of the views of others in the problem 
situation. 

• Identification of pre-existing solution (s) embedded in 
the initial problem situation. 

(III) Reframing of the problem (if necessary) . 

Descriptors : 

• Restatement of the problem in solution-free terms. 

• Identification and reliance on personal values related 
to a problem-solving process in restating the problem. 

• Anticipation of obstacles likely to arise during the 
problem-solving process. 

• Anticipation of ways to address obstacles should they 
arise . 
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TABLE 1 

Quasi-experimental Design: 

Descriptive Statistics - Cohort by Grand Mean of Problem- 
framing Ability 



Variable 


GrandMean 


SD 


Variance 


Cases 


Cohort 1.00 

(First year cohort - 
No exposure to PBL) 


1.0067 


.1247 


.0156 


6 


Cohort 2.00 


1.7500 


.2286 


.0522 


6 


(Second year cohort- 
1+ year exposure to 


PBL) 








Cohort 3.00 


2.2100 


.2463 


.0607 


6 


(Third year cohort- 
2+ years exposure to 


PBL) 








Entire Population 


1.6556 


.5459 


.2980 


18 
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TABLE 3 



Analysis of Covariance Summary Table 
Problem- framing Ability by Cohort with Covariate 


GREAQ 


Source 


Sum of 
Squares 


df 


Mean 

Square 


f Ratio 


Sig 
of f 


Main Effects 


4.908 


3 


1.636 


62.040 


.000 


COHORT 


2.399 


2 


1.199 


45.486 


.000 


GREAQ (Covar) 


.401 


1 


.401 


15.218 


.002 


Explained 


4.908 


3 


1.636 


62.040 


.000 


Residual 


.369 


14 


.026 






Total 


5.277 


17 


.310 







EXPERIMENTAL sums of squares 
Covariates entered WITH main effects 
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